Aiming at the problem that the existing neural network models have insufficient ability to characterize the features of classification objects in image classification tasks and cannot achieve high recognition accuracy, an image classification algorithm based on Lightweight Group-wise Attention Module (LGAM) was proposed. The proposed module reconstructed the feature maps from the channel and space of the input feature maps. Firstly, the input feature maps were grouped along the channel direction, and channel attention weight corresponding to each group was generated. At the same time, ladder type structure was used to solve the problem that the information between the groups was not circulated. Secondly, the global spatial attention weight was generated based on the new feature maps concatenated by each group, and the reconstructed feature maps were obtained by weighting the two attention weights. Finally, the reconstructed feature maps were merged with the input feature maps to generate the enhanced feature maps. Experiments were performed on the Cifar10 and Cifar100 datasets and part of the ImageNet2012 dataset with using the classification Top-1 error rate as the evaluation indicator to compare the ResNet, Wide-ResNet and ResNeXt enhanced by LGAM. Experimental results show that the Top-1 error rates of the neural network models enhanced by LGAM are 1 to 2 percentage points lower than those of the models before enhancing. LGAM can improve the feature characterization ability of existing neural network models, thus improving the recognition accuracy of image classification.